Direct Observation Of Clinical Skills Research Articles

Carle Illinois College of Medicine.

Medical Education Program Highlights Engineering integration: Carle Illinois College of Medicine (CIMED), a public–private partnership between the University of Illinois at Urbana–Champaign and the Carle Foundation, aims to graduate physician–innovators. As the backbone of this reenvisioned medical school, CIMED built engineering into every part of its 4-year, 3-phase curriculum. In Phase 1 preclinical courses, engineering concepts weave through weekly problem-based learning (PBL) cases and Medical Engineering Discovery and Innovation (MEDI) laboratory sessions. In Phase 2 clinical clerkships, an “engineering partner” faculty member rounds with students weekly to identify problems with potential engineering/technology/design solutions, which students then develop and address in the Innovation, Design, Engineering and Analysis (IDEA) course. Capstone and Data Science Project courses are part of Phase 3. Early clinical exposure: Clinical exposure begins immediately as students attend weekly sessions with a community-based preceptor for the first year of medical school. A longitudinal family medicine clerkship begins in July to start year 2. During the clerkship, students are the primary providers with faculty oversight in an evening clinic. The students have clinic sessions weekly for 18 months and receive weekly feedback from direct observation of clinical skills via a phone app. Innovative admissions process: Quantitative competencies must be evident from transcripts or workplace experiences. No on-campus interview is required for admission. Students produce a portfolio reflecting CIMED’s “4Cs”: competence, curiosity, compassion, and creativity. A video interview probes other CIMED values including collaboration and diversity. Carle Illinois’ first class matriculated in July 2018. Frequent and varied assessment: The medical education program at CIMED provides frequent opportunities for student retrieval practice. Throughout Phase 1 of the curriculum, students take weekly quizzes that assess their medical knowledge and receive immediate feedback. Team-based learning (TBL) is used in courses throughout Phase 1, which provides opportunities for individual retrieval and team discussion. Phase 1 assessment also includes 42 OSCE stations, 4 anatomy lab practicals, and a series of 5 progress exams that are designed to support spaced retrieval. In Phase 2 of the curriculum, a progress OSCE program is used to assess student competencies that are linked to Entrustable Professional Activities. Curriculum Curriculum description The CIMED curriculum consists of 3 phases in 4 years: an 18-month preclinical Phase 1; a 12-month clinical clerkship Phase 2; and Capstone and Data Science Projects, clinical electives, and career development in Phase 3. See Supplemental Digital Appendix 1—Curriculum Description—at https://links.lww.com/ACADMED/A831. Assessment See Supplemental Digital Appendix 2—Program Objectives—at https://links.lww.com/ACADMED/A831. Medical education program objectives Carle Illinois chose to adopt objectives combining 3 different frameworks: the AAMC Learning Objectives for Medical Student Education, ABET, Inc. Student Learning Outcomes, and KEEN (Kern Entrepreneurial Engineering Network) educational outcomes. It combined these frameworks to adopt objectives that aligned with the college’s mission and core competencies. In particular, the college added objectives that reflected engineering and innovation learning goals for physician–innovators. ABET is an accreditation agency that accredits engineering, computing, technology, and applied science degrees. ABET student outcomes are required for all engineering degrees and professional licensure as an engineer. KEEN educational outcomes are also widely used in engineering educational programs and help reinforce application of engineering and design skills introduced in the ABET objectives, as well as foster innovation. Pedagogy PBL: Carle Illinois students participate in three 2-hour PBL sessions per week during Phase 1. These sessions focus on cases that integrate foundational science, clinical practice, and engineering skills. Unique to CIMED is the addition of an “innovator” role; one student identifies a clinically relevant need from the case and leads a discussion of possible novel solutions. A point of pride at CIMED is having full-time faculty who facilitate the PBL sessions, providing continuity across Phase 1. TBL: While the primary pedagogy at CIMED is PBL, additional practices are used to build and solidify content and skill knowledge. These practices take the form of journal clubs, TBL, case review, and lecture. TBL is used often in the first course and then averages 1 to 2 TBL sessions per course for the remainder of Phase 1. Laboratory: In addition to traditional cadaver-based anatomy laboratories, students experience MEDI laboratory sessions demonstrating the application of engineering principles in medically relevant settings. For example, aerosol particle size and distribution patterns were reviewed in the setting of a lung model replicating asthma physiology. Immersive projects: In the latter portion of Phase 1, the MEDI sessions include group projects. In the Phase 2 IDEA Projects course, students identify patient and health care system problems and propose technological solutions. In Phase 3, one of the IDEA projects is selected as the student’s capstone project; this is pursued in an interdisciplinary team through the prototyping stage of development. The students also complete a data science project in Phase 3, analyzing large data sets for application to human health concerns. Simulation and virtual reality: Standardized patients, task trainers, high-fidelity manikins, and virtual/augmented reality activities are woven into coursework. Ultrasound training includes exploration of the engineering principles as well as the acquisition and interpretation of images. Students are tested to competence in ultrasound and other procedural skills. Clinical experiences Carle Illinois students do most of their training in community-based care settings. The primary hospital where students train is Carle Foundation Hospital, a 433-bed, Level I Trauma Center. In addition to the 22 outpatient facilities that are associated with Carle Hospital, students rotate through a local physician-owned clinic system, as well as a 210-bed private hospital. In the first year, students benefit from a yearlong course focused on clinical integration. Through didactics and simulation, students progressively develop the skills necessary to engage in the clinical environment. Students visit a dedicated preceptor in the community who assesses the learner’s development of their weekly skill. Carle Illinois students benefit from direct observation and timely, focused feedback. Required longitudinal experiences Longitudinal experiences are woven into 2 of the 3 curricular phases at CIMED. In year 1 of Phase 1, students are enrolled in the Introduction to Clinical Practice course, a yearlong course that focuses on the development of clinical skills; the course includes weekly sessions in a community physician’s practice. Starting in year 2 of Phase 1, students begin a 6-month family medicine clerkship. This clerkship includes ambulatory family medicine rotations, as well as a weekly student clinic. The student clinic continues through year 3 of Phase 2. Required and elective community-based rotations Phase 1 includes a 6-week elective period titled “Discovery,” where students have multiple options, with many selecting global experiences. Carle Illinois students have 7 required clerkships: family medicine, internal medicine, surgery, psychiatry, neurology, obstetrics–gynecology, and pediatrics. Family medicine is community based, and the psychiatry, internal medicine, and obstetrics–gynecology clerkships have portions that may occur in community locations outside of Carle Foundation Hospital. A total of 28 weeks of elective experience is required for graduation; many are community based or have portions that occur at community sites. Challenges in designing and implementing clinical experiences for medical students Similar to other new medical schools, CIMED faced some hurdles in the development of its clinical experiences. With a robust clinical immersion program in the first year, as well as a longitudinal student clinic that runs year-round, recruiting and onboarding new clinical faculty was a challenge. Additionally, Urbana–Champaign is a regional campus for the University of Illinois College of Medicine through 2022 and the needs of learners from both institutions must be balanced. Curricular Governance Curriculum governance committees: The Curriculum Oversight Committee is true to the CIMED mission with one-third of members from engineering; one-third of members from basic science; and one-third of members with clinical backgrounds, none of which hold an administrative title in the college. The committee sees and participates in review and approval of all courses throughout the 3 phases. To facilitate efficient review, the committee is split into 4 subcommittees: the Preclerkship Subcommittee, Clerkship Subcommittee, Engineering Subcommittee, and Program Evaluation Subcommittee. Each subcommittee does detailed review of proposals and then presents its review for endorsement of the larger committee. In addition, any administrator, faculty, student, or staff member in the college is welcome to submit materials for committee input or attend any of the public meetings. This process ensures that faculty control the curriculum without influence from the administration and that all stakeholders have equal access to the committee to submit proposals. Department-led governance: As a public–private partnership, CIMED has a unique faculty model. As in many schools, the clinical faculty are employed by the health system (Carle Foundation or other community partners) and have appointments at CIMED; key personnel such as clerkship directors and course directors have designated FTE contributions to the school. Carle Illinois follows a similar system for its foundational science and engineering faculty; the faculty retain appointments in their home college, and course directors have designated effort in CIMED. The college draws on the wealth of content expertise on the University of Illinois campus. There are two departments within CIMED: Clinical Sciences, and Biomedical and Translational Sciences; the department chairs oversee faculty efforts dedicated to CIMED and report to the dean. Each of the departments maintains an Appointment, Promotions, and Tenure Committee and accesses shared support services at the college level. See Figure 1—Curriculum Oversight Committee.Figure 1: Curriculum Oversight Committee.See Figure 2—Organizational chart.Figure 2: Organizational chart.Education Staff Course design: All Phase 1 courses are designed by a minimum of 3 faculty: a basic scientist, clinician, and engineer. Academic affairs staff support all facets of curriculum development and delivery, including management of the supporting software, testing programs, and course delivery. An academic affairs faculty or staff member is appointed to each course committee. Roles of academic staff: Carle Illinois has various roles to fully support the curriculum. These include: medical education facilitators (PBL); directors for curriculum, faculty development, and assessment and quality improvement; academic affairs coordinator; medical education coordinators (clinical, basic science); medical librarians; simulation center staff (manager, equipment technician, simulated patient coordinator); engineering/IDEA/data science/capstone coordinator; and clerkship coordinators. These staff work with the faculty to ensure the curriculum is created, developed, and delivered according to the values and mission of CIMED. Faculty Development and Support in Education Professional development for faculty as educators: There are a variety of options at CIMED for faculty professional development. Specific training is available for all roles: facilitators, clinical instructors, engineers, and administrative staff. There is an introductory presentation titled “Introduction to Carle Illinois—What Makes CIMED Unique.” An annual curriculum retreat focuses on curricular updates, learning new skills, and network building. It features a mix of presentations, active learning sessions, and workshops. Another annual event, the Problem-Based Learning Workshop, allows all faculty and staff to experience PBL as both a student and facilitator. Faculty may request one-on-one sessions for assistance with tasks such as creating learning objectives, micro-lecture recordings, active learning activities, and so forth. Additionally, the University of Illinois at Urbana–Champaign has extensive faculty development offerings. Promotion and tenure: To support both the clinicians and campus faculty, in addition to the faculty with defined FTE roles, there are various tracks for appointment, promotion, and tenure. These include teaching, clinical, and tenure tracks. In each of these tracks, student evaluation and scholarship are included in consideration for promotion.

Teaching Diagnostic Reasoning to Faculty Using an Assessment for Learning Tool: Training the Trainer.

IntroductionThere is a need for a standardized approach to understand and assess clinical reasoning in medical learners. The Assessment of Reasoning Tool was developed based on prevalent theories and frameworks using a multidisciplinary expert panel. As the tool provides a standardized rubric for assessing clinical reasoning, we designed an interactive train-the-trainer workshop for clinical educators and education leaders interested in improving their teaching skills and/or introducing curricula surrounding diagnostic reasoning.MethodsIn this workshop, participants were exposed to the major domains of diagnostic reasoning and how to apply it to the assessment of a learner's skills. Kolb's experiential learning was the underlying model, which we showcased by using multiple interactive techniques, including small-group discussion, peer sharing, and case practice. We presented the workshop at a national conference of pediatric educators and as a faculty development workshop at a single institution. Participants were asked to complete a survey after the workshop to gauge their reactions and look for areas of improvement.ResultsA total of 34 participants attended the two workshops. Participants rated the workshop favorably, with most planning to make a change to their practice. Comments were largely positive, emphasizing the benefits of the interactive approach.DiscussionThe workshop and teaching materials represent an important early step in the workplace-based assessment of diagnostic reasoning in medical learners. Grounded in the clinical reasoning literature, the workshop offers one approach to assessing these skills in learners with or without direct observation of clinical skills.

"All the ward's a stage": a qualitative study of the experience of direct observation of handoffs.

Direct observation of clinical skills is central to assessment in a competency-based medical education model, yet little is known about how direct observation is experienced by trainees and observers. The objective of the study is to explore how direct observation was experienced by residents and faculty in the context of the I-PASS Handoff Study. In this multi-center qualitative study, we conducted focus groups and semi-structured interviews of residents and faculty members at eight tertiary pediatric centers in North America that implemented the I-PASS Handoff Bundle. We employed qualitative thematic analysis to interpret the data. Barriers to and strategies for direct observation were described relating to the observer, trainee, and clinical environment. Residents and faculty described a mutual awareness that residents change their performance of handoffs when observed, in contrast to their usual behavior in a clinical setting. Changes in handoff performance may depend on the nature of the observer or 'audience'. Direct observation also highlighted the importance of handoffs to participants, recognized as a clinical activity that warrants feedback and assessment. Dramaturgical theory can be used to understand our finding of 'front-stage' (observed) versus 'backstage' (unobserved) handoffs as distinct performances, tailored to an "audience". Educators must be cognizant of changes in performance of routine clinical activities when using direct observation to assess clinical competence.

Experiences in DOCS Clinic: Direct Observation of Clinical Skills

PURPOSE: To investigate the efficacy of an assessment tool for core entrustable professional activities (EPAs) in obstetrics and gynecology. BACKGROUND: Medical education is moving toward competency-based assessment. EPAs standardize proficiency levels for learners and provide assessment frameworks, but require direct observation. Students report little direct observation of history taking or physical exam skills in clerkships. We implemented a standardized clinical experience and assessment tool to assess student progress on EPAs. METHODS: We developed an assessment tool for history, physical exam, and counseling, with behavioral anchors to describe achievement levels. Participants included OBGYN clerkship students in 2016-2017. Students conducted patient encounters with direct observation and feedback by faculty. Students then conducted a separate patient encounter with an experienced educator, who rated performance using the assessment tool. RESULTS: Forty-two of 63 students participated. 98% rated emerging or entrustable level in history-taking (EPA 1A) and physical exam (EPA 1B). Students who were assessed on differential diagnosis (EPA2), and ordering/interpreting tests (EPA 4) were rated pre-entrustable or emerging. All students demonstrated entrustability in collaboration and teamwork (EPA 9). DISCUSSION: Most students were rated emerging or entrustable in EPA 1; all demonstrated entrustability in EPA 9. Our assessment tool provided a framework to ensure consistency in assessment of EPAs. Our study was limited by availability of clinical educators. We plan to refine the assessment tool to develop standards for each EPA. Faculty development is required as clinical educators are not familiar with this framework. Change in the educational culture is needed for EPAs to be successful.

Guidelines: The do's, don'ts and don't knows of direct observation of clinical skills in medical education.

IntroductionDirect observation of clinical skills is a key assessment strategy in competency-based medical education. The guidelines presented in this paper synthesize the literature on direct observation of clinical skills. The goal is to provide a practical list of Do’s, Don’ts and Don’t Knows about direct observation for supervisors who teach learners in the clinical setting and for educational leaders who are responsible for clinical training programs.MethodsWe built consensus through an iterative approach in which each author, based on their medical education and research knowledge and expertise, independently developed a list of Do’s, Don’ts, and Don’t Knows about direct observation of clinical skills. Lists were compiled, discussed and revised. We then sought and compiled evidence to support each guideline and determine the strength of each guideline.ResultsA final set of 33 Do’s, Don’ts and Don’t Knows is presented along with a summary of evidence for each guideline. Guidelines focus on two groups: individual supervisors and the educational leaders responsible for clinical training programs. Guidelines address recommendations for how to focus direct observation, select an assessment tool, promote high quality assessments, conduct rater training, and create a learning culture conducive to direct observation.ConclusionsHigh frequency, high quality direct observation of clinical skills can be challenging. These guidelines offer important evidence-based Do’s and Don’ts that can help improve the frequency and quality of direct observation. Improving direct observation requires focus not just on individual supervisors and their learners, but also on the organizations and cultures in which they work and train. Additional research to address the Don’t Knows can help educators realize the full potential of direct observation in competency-based education.

Formative Assessment by Direct Observation of Clinical Skills on the OB/GYN Clerkship Rotation

Local Mentor: Margaret Burnett, MD, FRCSC APGO Advisor: John Dalrymple, MD PURPOSE: To determine the impact of requiring that medical students be observed taking a history or performing a pelvic examination by a physician during their OB/GYN clerkship rotation. BACKGROUND: One of the LCME standards for accreditation of a medical school is having physicians directly observe medical students in their clinical activities in order to provide formative feedback. Since 2009, 29.7–52.9% of each class of Manitoba medical graduates report having been observed taking a history on their OB/GYN clerkship rotation; 45.2–70.4% for observed physical examination. METHOD: Clerks are surveyed on the last day of their OB/GYN rotation to determine the number of observed histories and/or pelvic examinations. In September 2015, observed histories and pelvic examinations became a rotation requirement. A mini-CEX tool was developed and included in the student's logbook to facilitate documentation and provide feedback. The reported number of observed histories and pelvic examinations are compared before and after implementation of this process. RESULTS: Thirty-five percent of 69 students surveyed before implementation report having been observed performing a history by a physician. This rose to 64% (of 28 students) following implementation. All of them report being observed performing a pelvic examination, before and after, however only 53% documented this on the assessment tool. DISCUSSION: The number of observed histories has increased however there is room for further improvement. This may require additional faculty education and/or scheduled assessment for some students. The number reporting an observed pelvic examination continues to be 100% but improved documentation of this must be encouraged.

Feedback in the OSCE: What Do Residents Remember?

Theory: The move to competency-based education has heightened the importance of direct observation of clinical skills and effective feedback. The Objective Structured Clinical Examination (OSCE) is widely used for assessment and affords an opportunity for both direct observation and feedback to occur simultaneously. For feedback to be effective, it should include direct observation, assessment of performance, provision of feedback, reflection, decision making, and use of feedback for learning and change. Hypotheses: If one of the goals of feedback is to engage students to think about their performance (i.e., reflection), it would seem imperative that they can recall this feedback both immediately and into the future. This study explores recall of feedback in the context of an OSCE. Specifically, the purpose of this study was to (a) determine the amount and the accuracy of feedback that trainees remember immediately after an OSCE, as well as 1 month later, and (b) assess whether prompting immediate recall improved delayed recall. Methods: Internal medicine residents received 2 minutes of verbal feedback from physician examiners in the context of an OSCE. The feedback was audio-recorded and later transcribed. Residents were randomly allocated to the immediate recall group (immediate-RG; n = 10) or the delayed recall group (delayed-RG; n = 8). The immediate-RG completed a questionnaire prompting recall of feedback received immediately after the OSCE, and then again 1 month later. The delayed-RG completed a questionnaire only 1 month after the OSCE. The total number and accuracy of feedback points provided by examiners were compared to the points recalled by residents. Results comparing recall at 1 month between the immediate-RG and the delayed-RG were also studied. Results: Physician examiners provided considerably more feedback points (M = 16.3) than the residents recalled immediately after the OSCE (M = 2.61, p < .001). There was no significant difference between the number of feedback points recalled upon completion of the OSCE (2.61) compared to 1 month later (M = 1.96, p = .06, Cohen's d = .70). Prompting immediate recall did not improve later recall. The mean accuracy score for feedback recall immediately after the OSCE was 4.3/9 or “somewhat representative,” and at 1 month the score dropped to 3.5/9 or “not representative” (ns). Conclusion: Residents recall very few feedback points immediately after the OSCE and 1 month later. The feedback points that are recalled are neither very accurate nor representative of the feedback actually provided.

How faculty members experience workplace-based assessment rater training: a qualitative study.

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training. This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and amodified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed. In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data. Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in. Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

Systematic Direct Observation of Clinical Skills in the Clinical Year

Abstract Introduction Direct observation of clinical skills is vital to medical student development. However, direct observation of medical students by faculty is often sporadic, undocumented, and ...

Not Just Good Pizza and Bad Politics: The Chicagoland Solution to our Workforce Shortage (SA505)

1.Describe the creation and integration of a novel multicentered, longitudinal education and mentoring program for interdisciplinary providers: the Coleman Palliative Medicine Training Program.2.Demonstrate the use of individual practice improvement projects to create meaningful, sustainable changes in palliative care services.3.Practice the application of Intent to Change tools used in our program to engage learners and assist with goalsetting, project development, and mentorship. The growth of palliative medicine programs is dramatically outpacing the number of providers who are adequately trained to provide high-quality care to patients and families living with serious illness. In response, hospice and palliative medicine leaders across the greater Chicagoland area, with support from a local foundation, created the Coleman Palliative Medicine Training Program. This 2-year longitudinal program is currently training 35 doctors and nurses from 23 hospitals seeking to start or improve existing palliative medicine programs. During this session, we will describe essential components of this large-scale multicentered educational program, including recruitment tactics, curriculum development, and evaluation. Methods of teaching and community building will be highlighted, including e-learning through an open-access website, biannual workshops, and direct observation of clinical skills with assigned mentors. One unique feature of this program involves the use of Intent to Change tools centered on learners' developing, implementing, and evaluating a practice improvement project to improve the quality of or access to primary palliative care services at their respective practice settings. Twenty-two mentors from 10 academic palliative medicine and hospice organizations provide direct observation opportunities and guidance around the implementation and evaluation of these projects throughout the 2-year program. During the interactive portion of this session, participants will review examples of practice improvement projects and employ the use of Intent to Change tools as a potential method to enhance interprofessional development at their home institution.

Standardizing the Pre-Licensure Supervision Process: A Commentary on Advocating for Direct Observation of Skills

The present paper advocates for standardized regulations and laws for supervision of pre-licensed counselors in the United States, particularly for direct observation of clinical skills. A review of regulations by the American Counseling Association's (ACA) Professional Affairs Division (2012) reveals that only two states (Arizona and North Carolina) specify requiring supervision interventions that include the use of reviewing audio or videotapes, or live supervision modalities, to help evaluate pre-licensed counselors competence. Literature on the current state of regulations as well as extant research in supervision practices, and effects on satisfaction with supervision and self-efficacy is presented and framed in terms of standardized policy.Keywords: pre-licensed counselors, regulations, laws, satisfaction with supervision, self-efficacy, direct observationA review of regulations by the American Counseling Association (ACA) Office of Professional Affairs (2012) reveals that all fifty states require the practice of counseling under supervision for two or more years after the completion of the master's degree prior to licensure. However, there are no unified national standards that govern post-master's degree supervision (e.g., ACA, 2012 Borders & Cashwell, 1992). Supervised counseling experience ranges from 1000 to 4500 hours (ACA Office of Professional Affairs), with the amount of face-to-face clinical supervision occurring either in an individual or group setting varying greatly (ACA, 2007). Furthermore, only two states (Arizona and North Carolina) specify requiring supervision interventions that include the use of reviewing audio or videotapes, or live supervision modalities to help evaluate pre-licensed counselor competence (ACA Professional Affairs Division). Some studies have investigated supervision in this counselor pre-licensure stage, including factors related to counselor effectiveness such as self-efficacy and satisfaction (e.g., Bernard & Goodyear, 2004; Fall & Sutton, 2004; Magnuson, Norem, & Wilcoxon, 2002). The use of direct observation of skills in supervision is noted to lead to more positive effects in terms of counseling performance and outcomes. Standardization of required direct observation of clinical skills is especially necessary. The literature indicates that observation of skills is crucial to counselor professional development and practice (Herbert & Trusty, 2006). Moreover, standardization of required direct observation of clinical skills is a key factor in licensure policy The following review calls for such standardization, with a summary of current regulations, extant research in counseling outcomes and supervision practice, and recommendations and advocacy for regulation.Supervision Background and StandardsACA, the Association for Counselor Education and Supervision (ACES), and the American Association of State Counseling Licensure Boards (AASCB) have all attempted to define best practices in counselor supervision. The ACA Code of Ethics standard F. 1 .a, discusses a primary obligation of supervisors in the role of monitoring services of counselors-Neal in-training. This monitoring consists of case notes, samples of clinical work or live observation of the trainee (Herlihy & Corey, 2006). In 1990, model legislation for licensed professional counselors was developed by the American Counseling Association to promote acceptable professional standards within the realm of counseling. This proposed licensure bill recommended state licensure boards consider what is the nature of the supervision co-therapy, direct observation, audio and/or videotaping (Bloom et al., p. 520). ACES also addressed this issue in two different documents. First, the Ethical Guidelines for Counseling Supervisors states that supervisors are responsible for actual work samples via audio or videotape or live observation... which should be reviewed by the supervisor as a regular part of the ongoing supervisory process (ACES, 1995, p. …

78 Direct Observation: There Is an App for That!

Direct observation of clinical skills is a critical step in the creation of an accurate assessment of performance. A portfolio of observations can be useful in appreciating learner strengths and weaknesses.

Faculty staff perceptions of feedback to residents after direct observation of clinical skills

Performance-based workplace assessments are increasingly important in clinical training. Given the inaccuracy of self-assessment, the provision of external feedback to residents about their clinical skills is necessary for the development of expertise. However, little is known about the processes used by faculty members in giving feedback to residents after observing them with patients. This study explores the factors that underpin faculty members' decisions regarding the feedback they give to residents after directly observing them with patients and the factors that influence how feedback is delivered. In 2009, 44 general internal medicine faculty staff responsible for out-patient resident teaching from 16 internal medicine residency programmes watched four videotaped scenarios and two live scenarios of standardised residents (SRs) with standardised patients and rated the SRs using the mini-clinical evaluation exercise (mini-CEX) format. Faculty staff also provided feedback to the SRs after the live encounters. After each encounter, faculty staff were individually interviewed using a semi-structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods. Two broad themes were identified in faculty members' descriptions of the feedback process: variability in feedback techniques, and the factors that influence how faculty staff think and feel about delivering feedback. Multiple approaches to feedback delivery were observed. Faculty members' tensions in balancing positive and negative feedback, their own perceived self-efficacy, their perceptions of the resident's insight, receptivity, skill and potential, the faculty member-resident relationship and contextual factors impacted the feedback process. The provision of feedback by faculty staff to residents after observing resident-patient interactions is a complex and dynamic process and is influenced by many factors. Understanding these cognitive and affective factors may provide insight into potential new approaches to faculty development to improve faculty staff's feedback skills and the effectiveness of their feedback.

Development of a longitudinal integrated clerkship at an academic medical center

In 2005, medical educators at the University of California, San Francisco (UCSF), began developing the Parnassus Integrated Student Clinical Experiences (PISCES) program, a year-long longitudinal integrated clerkship at its academic medical center. The principles guiding this new clerkship were continuity with faculty preceptors, patients, and peers; a developmentally progressive curriculum with an emphasis on interdisciplinary teaching; and exposure to undiagnosed illness in acute and chronic care settings. Innovative elements included quarterly student evaluation sessions with all preceptors together, peer-to-peer evaluation, and oversight advising with an assigned faculty member. PISCES launched with eight medical students for the 2007/2008 academic year and expanded to 15 students for 2008/2009. Compared to UCSF's traditional core clerkships, evaluations from PISCES indicated significantly higher student satisfaction with faculty teaching, formal didactics, direct observation of clinical skills, and feedback. Student performance on discipline-specific examinations and United States Medical Licensing Examination step 2 CK was equivalent to and on standardized patient examinations was slightly superior to that of traditional peers. Participants’ career interests ranged from primary care to surgical subspecialties. These results demonstrate that a longitudinal integrated clerkship can be implemented successfully at a tertiary care academic medical center.

Weill Cornell Medical College of Cornell University

Curriculum Management and Governance Structure ♦ The governing body of the curriculum is the Medical Education Council (MEC), a standing committee of the medical college, which is chaired by the Senior Associate Dean (Education) and whose members are appointed by the Dean. (See Figure 1.)FIGURE 1:: Curriculum Management♦ The MEC meets on a monthly basis and has six subcommittees: Core Basic Science Committee (CBSC), Advanced Biomedical Science Committee (ABS), Clinical Curriculum Committee (CCC), Curriculum and Faculty Evaluation Committee (CFEC), Admissions Committee, and the Promotion and Graduation (P&G) Committee, the chairs of which are appointed by the Dean. ♦ Each subcommittee has a specific charge to monitor and propose changes within its purview. The MEC and all of the subcommittees except for the P&G Committee have student representatives. ♦ The membership of the MEC, comprising senior faculty and students who play leadership roles in the educational programs, is not a departmentally based representation, a distinct feature that enables it to serve the interests of an interdisciplinary curriculum most fairly. ♦ The 16 voting members of the MEC are the chairs of the CBSC and the ABS, the chair and vice-chair of the CCC and the CFEC, four representatives from each of the faculty leadership councils (Executive Faculty Council and General Faculty Council), and two student representatives. ♦ The MEC has operational responsibility for the planning, implementation, and monitoring of the curriculum and serves as the steering committee for all policy decisions involving the undergraduate medical educational programs of the WCMC. ♦ All major educational policy decisions and curriculum changes must be approved by the MEC. These decisions are informed by information from the subcommittees, the course design groups, and student representatives. Office of Education ♦ A division under the Office of Academic Affairs, the Office of Curriculum and Educational Development (OCED), whose director is the Associate Dean for Curricular Affairs and whose full-time associate director is an educational specialist, provides administrative and logistic support for all basic science courses, the Medicine, Patients, and Society (MPS) courses in all three years, and the Primary Care clerkship. ♦ OCED also provides centralized functions for the clinical clerkships, such as course evaluation and curricular mapping. ♦ OCED, with a full-time staff of ten, provides guidance to faculty on curricular content and design, conducts faculty development workshops, monitors student assessment in courses, conducts centralized electronic course evaluation surveys of students, administers course budgets for the centrally governed courses, and sponsors the yearly Education Retreat. Financial Management of Educational Programs ♦ WCMC, with only 12% of revenues coming from investment income (endowments), weathered the financial crisis through a careful analysis of potential spending reductions and by implementing conservative financial management strategies such as temporizing faculty and staff raises and cutting staff positions where possible. ♦ A WCMC Task Force cost containment program will reduce the general operating budget by almost 10% in 2010. ♦ Additional revenue declines are still expected in the near future, but the educational programs and student financial aid is currently being supported from central funding sources. ♦ WCMC's largest revenue producer, the clinical enterprise (Physician Organization), has remained on budget during this time due to the diversity of services provided and the patient base demographics. Valuing Teaching ♦ The Medical College makes an effort to recognize exceptional teachers through annual teaching awards. ♦ The Senior List at Convocation consists of recognition by the Senior Class of 16 individuals who were judged to be outstanding teachers. ♦ There is an annual Excellence in Teaching awards dinner, where 20+ faculty are recognized for their teaching contributions each year. ♦ The teaching of medical students is an important criterion used in the promotion process for all tracks. Evidence includes time spent in teaching (i.e., Course Directorship) and excellence in teaching (i.e., Teaching Awards, Scholarship). ♦ On the clinical promotion track, excellent teachers have been promoted based largely on that basis. ♦ There is a newly created academic-educator track for academic appointment and promotion where faculty who are engaged primarily in educational activities with little or no obligations in research or professional/clinical service and who demonstrate a record of excellence in education may qualify for appointment or promotion. ♦ There are currently 24 faculty (14 male, 10 female) on the academic-educator track: 8 professors, 7 associate professors, and 9 assistant professors. ♦ In 2007, the Dean accepted Task Force recommendations specifying faculty members' contribution to medical student education (40 hours/year), the compensation of education leadership, procedures in the recruitment of faculty to the teaching mission, and the transition of course leadership. ♦ Another major initiative from the Dean's office is the annual Departmental Teaching Plans. Beginning in 2005, the general budget support to each department from the Medical College included an amount based upon the teaching provided by the faculty from that department. ♦ While the global financial situation has mitigated the ability to continue to expand new monies to the departments, the Departmental Teaching Plan process stipulates that 25% of the departmental budgets are not released each year until the department chair signs a negotiated Teaching Plan with the Senior Associate Dean (Education) that assures adequate and appropriate faculty staffing of all required courses, evidence of the importance that the institution places on education. ♦ Current requirements of the Teaching Plans include providing discipline-specific faculty expertise to support the educational program, protecting the time of course directors, providing for associate course directors, and creating a Departmental Teaching Database documenting the teaching activities of each of the full-time faculty members in the department who teach in required courses of the curriculum. ♦ As a component of the most recent Strategic Plan, fundraising efforts were included for ten Education Scholars. ♦ The plan is to raise money to fund endowments, each at $1 million, to provide substantive support for faculty members to engage in education scholarship (e.g., develop new curriculum or new curricular resources; pilot new learning formats or assessment methods). It is expected that fundraising will begin in the current academic year. Curriculum Renewal Process ♦ Curriculum renewal is an ongoing process at WCMC as evidenced by the new topics added to the curriculum, new pedagogies, and new assessments (see below). ♦ The curriculum is reviewed periodically by the Curriculum and Faculty Evaluation Committee with particular attention to outcome performance measures of our students and graduates. ♦ The recently completed Institutional Self-Study for the LCME accreditation process has identified areas in which new curriculum formats and objectives will be developed in the coming year. ♦ This curriculum reform effort will investigate new pedagogies, course content and sequencing, new programmatic initiatives, new assessment strategies focused on competencies, and an enhancement of faculty development. Learning Outcomes/Competencies The Objectives for the Educational Program are available at: http://weill.cornell.edu/education/curriculum/edu_obj.html. ♦ The focus of our upcoming curriculum reform will include enhancing competency-based assessment. New Topics in the Curriculum Since 2000 ♦ “Patient safety” and “quality improvement” are taught in the WCMC curriculum Lecture-seminar in Public Health Clerkship, “Medico-Legal Issues in Health Care” Lecture-seminar in Host Defenses, “How Are Drugs Approved?” and “Drug Interactions in Patients” Lecture-seminar in Introductory Clerkship, “Universal Precautions” and “Introduction to Medical Informatics” Lecture-seminar in Medicine, Patients, and Society 2, “Health Care Reform Issues in Quality Assurance” and “Hospital Health and Safety” Lecture-seminar in Psychiatry, “The Suicidal Patient” Lecture-seminar in Medicine, Patients and Society (MPS) 1, “Mistakes in Medicine” ♦ Simulations/training in new surgical techniques New procedures curriculum in Introductory Clerkship (IC): Suturing workshops/lab for all students, EKG lab (students learn how to take EKGs not just read them), video demos of common procedures such as Foley catheter insertion IC: Students use a simulator (mannequin) in a chest pain/myocardial infarction case to learn how to handle acute emergencies. ♦ A four-year grant from the Donald W. Reynolds Foundation has enabled the integration of geriatrics into all four years of the medical student curriculum beginning with three four-hour educational interventions in MPS I: “Introduction to the Geriatric Patient” module: Film, drama, and “live patients” are used to discuss ageism, effective communication skills, managing the third person in the interview, the enhanced social history, and functional assessment. The second module exposes students to the experience of living with chronic illness from the perspectives of the patient, family, and interdisciplinary team. The third module, “Functional Assessment, Disability, and Rehabilitation,” requires students to navigate the hospital in a wheelchair, performing assigned “tasks” to heighten their awareness of the challenges that disabled patients face every day. Geriatrics has also been integrated into PBL cases and standardized patient exercises in Years 1–2, and a special session on the geriatrics exam has been added to MPS II. The Primary Care Clerkship provides geriatrics service-learning experiences for all students, including rotations at clinical geriatrics sites, blood pressure screening at a senior center, house visits with geriatrics faculty, lectures on problems common in geriatrics patients, and in a multimedia course called Environmental Geriatrics, created at WCMC, where students enter a virtual home of patients with functional loss, viewing animated patients encountering remediable environmental hazards that increase fall and burn risk. Through sound, text, and moving images, learners witness virtual patients who have reduced visual, hearing, mobility, and memory skills struggling to accomplish tasks of daily living. In the Surgery Clerkship, students receive a lecture on “The Perioperative Care of the Older Adult.” In the Medicine Clerkship, students participate in “Fast-Forward Rounds” that foster awareness of the essential elements of transitional care. In Year 4, a four-week elective in geriatrics can be individualized based on the student's interest, such as palliative care, acute care, ambulatory care, and geriatric psychiatry. ♦ Oates' Communication Skills Curriculum The Oates' Communication Skills Curriculum (named for an alumnus of WCMC) is a thematic series of communication skills sessions and exercises integrated throughout the four years of the educational program. Year 1, in MPS I, the emphasis is on communicating with patients and their families. It includes sessions related to general communication skills, breaking bad news, educating patients, and improving adherence. The curriculum includes skills group exercises, exercises in the office preceptor experience, and standardized patient experiences. Year 2 continues with exercises and standardized patient experiences in history taking and physical examination. In Years 3–4, the Medicine and Pediatrics clerkships include group OSCE sessions that emphasize nuances in communicating to patients and their families through role-play and other active learning formats. ♦ Medicine, Patients, and Society I curriculum enhancements Since 2000, a number of enhancements have been made to the MPS courses to address evolving competencies. These included: (1) Expanded cultural competence series: Medicine and Culture, Patient-Based Cross Cultural Care, Health Care Disparities, Interpreter Skills; (2) New lectures on History of Medicine, Introduction to the Academic Medical Center, Health and Human Rights, Health Care Delivery in Resource Poor Settings, Vital Signs, Illness and Disease, Clinical Reasoning; (3) New Communication skills sessions including Communication Skills and Clinical Reasoning, Sharing Bad News, Adherence and Patient Education, The Patient's Context and the Social History, Smoking Cessation; (4) New Life-Cycle lectures: Birth, Geriatrics (Communicating with the Older Adult), Palliative Care, Chronic Illness; (5) First Responder course and BCLS—combination of lectures and workshops; (6) New lectures on taking a Diet History, Nutrition, and Global Health; and (7) Increased emphasis on linkage between lecture/seminar topics and Office Preceptor (OP) sessions. Changes in Pedagogy ♦ In addition to the major shift of our curriculum to Problem-Based Learning (PBL) and small group teaching in 1996, the following additions to our repertoire of pedagogy have been made: Group OSCE in Oates' Communication Skills Curriculum. Simulator sessions in Human Structure & Function Course to teach valvular dysfunction and cardiac pathophysiology; the first responder reaction during orientation; and extracurricular simulator sessions for students on an ad hoc basis. Standardized patients to teach the neurological examination in MPS II, longitudinal clinical skills curriculum using standardized patients, and videotape review for history taking, physical examination, and patient education. Trips to the Metropolitan Opera (Medicine and Humanities) to view the role of medicine in classical opera. Web-based Self-Assessment Quizzes: Many courses and clerkships now have web-based self-assessment quizzes. Panel discussions that bring experts to discuss ethical decisions in medicine, how to handle the impaired physician, how to handle “mistakes” in medicine. PBL has recently replaced some didactic seminars in the OB/GYN clerkship, a change well received by students. Reflective Journals: In MPS I in which students reflect on their experiences and share them in small groups led by faculty facilitators each week, and in MPS III, students keep reflective diaries of their experiences on the pain and palliative care service of Memorial Sloan Kettering Cancer Center. Field trips in Public Health, in MPS III to Metropolitan Museum (cultural norms in death and dying) and to Calvary Hospital (hospice care). Role Plays (students playing role of doctor, patient, family member) in teaching venues to teach students how to conduct empathic interviews and experience the role of the patient. Changes in Assessment ♦ Each basic science course ends with a Triple Jump Examination held over two days. The first “jump” of the examination is a written exercise intended to assess baseline knowledge. The second “jump” consists of an afternoon and evening study period to prepare for an oral examination on the second day. The third “jump” is a 25-minute one-on-one oral examination with a faculty member in which the student may correct any misconceptions or knowledge gaps from the first part of the examination. The session is structured to assess both the student's factual knowledge and reasoning skills related to the case. ♦ Satisfaction among students with this examination format is extremely high because it emphasizes concepts and reasoning skills. ♦ There is a longitudinal clinical skills program in the WCMC Clinical Skills Center (CSC), allowing for direct observations of the progression of students' clinical skills and providing multiple opportunities for formative feedback on performance. ♦ The current simulated patient sessions required of each student form a longitudinal curriculum whose sessions include: Year 1: Complete medical history and integrated cardiac physiology (using SPs and a mannequin simulator) Year 2: Complete history and physical examination, neurological examination, and male genitourinary and gynecological examination (breast and pelvic exam) Years 3–4: Mid-clerkship OSCEs Year 4: Comprehensive Graduation OSCE ♦ The DOCS (Direct Observation of Clinical Skills) program provides students with mid-clerkship formative feedback on directly observed clinical skills. Faculty and residents observe students in interviewing/communication, physical examination, and oral presentation. ♦ Observers complete a form that includes a checklist of key clinical skill components and a section for formative feedback, which is discussed with the student. Clinical Experiences ♦ WCMC places a high priority on diversity in clinical experiences and students care for patients from an immense variety of social, ethnic, economic, religious, and cultural backgrounds using a network of clinical teaching sites that offers a wide range of learning opportunities with different patient populations. ♦ All clerkships utilize multiple clinical sites in different configurations. These sites include New York Presbyterian Hospital/Weill Cornell campus, New York Hospital Queens (NYH-Q), The Brooklyn Hospital Medical Center, Lincoln Hospital, Memorial Sloan Kettering Cancer Center, The Methodist Hospital (Houston, TX), Methodist Hospital (Brooklyn), New York Presbyterian Hospital (Westchester Division), and Cayuga Medical Center & Gannett Cornell University Student Health Service, Ithaca, NY. Challenges ♦ In MPS I, each student is paired with one physician for clinical tutorials in physicians' offices, requiring approximately 100 clinical positions. ♦ Although it has been challenging to supply the large number of sites required, the effort has been successful, and a relatively stable cadre of office preceptors has developed. ♦ The challenges at multiple network sites include managing the ways in which attendings and housestaff are deployed for teaching purposes. ♦ At NYPH, where residents are highly qualified, the residents are a major source of teaching. ♦ At affiliate hospitals, attending faculty members provide proportionately more teaching. Those residents at affiliate sites who do teach students are often selected specifically for their teaching abilities. ♦ Another challenge includes monitoring the educational quality at clinical sites in a variety of ways. ♦ Clerkship directors are in close contact with affiliate site directors on a regular basis, generally at the beginning of the clerkship rotations, and on an ad hoc basis as problems may arise. ♦ The clerkship directors regularly review the grade distribution of the students who rotate at affiliates to determine if there is any significant difference among the sites. Currently, the review of those data shows that student performance is similar at all of the clinical sites with respect to grades. ♦ The monitoring of learning environment and educational quality of the clinical sites is assisted by a centralized course evaluation process. Clerkship directors have real-time ongoing access to all student evaluations through the E-value system. ♦ In addition, the Curriculum Office (OCED) provides the clerkship directors with an annual summary of site evaluations, which allows them to compare their sites on various parameters such as the quality of attending and resident teaching, supervision, feedback, and patient case mix. Highlights of the Program/School ♦ In addition to the thematic integration of Geriatrics, the Oates' Communication Skills Curriculum, and the DOCS exercises described above, the following offerings are curricular and extracurricular highlights. ♦ Integrated interdisciplinary courses with early exposure to patients in the first year, progressive acquisition of clinical skills during first two years, an emphasis on problem-solving and critical-thinking skills, and multiple learning formats including problem-based learning, lectures, seminars, journal club, labs, small group tutorials, field trips, web-based programs, standardized patients, and patient demonstrations. ♦ MPS III, The Medicine, Patients, and Society (MPS) III course is a two-week required rotation devoted to clinical ethics, palliative care, pain management, and end-of-life decision-making. ♦ In addition, students follow terminally ill patients on clinical services (“participatory observation”) and, in a seminar format, critically discuss their observations of end-of-life care. The course is interdisciplinary, involving faculty from pastoral care, hospice care, the Memorial Sloan-Kettering Cancer Center, and other departments. ♦ The Public Health Course is a two-week required course that exposes students to health care delivery systems. Students engage in field trips to public health care delivery sites in New York City, meet with senior executives in health care delivery agencies, and participate in didactic and debate sessions on health care delivery issues. ♦ Research opportunities in basic biomedical sciences, clinical, and translational medicine, epidemiology and public health policy at major biomedical institutions in New York City and research training programs (Masters in Clinical Investigation) through the new Clinical Translational Science Center. ♦ State-of-the-art clinical skills center with standardized patients and simulators and state-of-the-art gross anatomy laboratory with full body dissection supplemented by prosections and models. ♦ Clinical experiences at nation's top tertiary care centers (New York Presbyterian Hospital; Memorial Sloan Kettering Cancer Center; Hospital for Special Surgery) and at community and municipal hospitals, nursing homes, and outpatient clinics in one of the most diverse cities in the world. Global Health Curriculum/International Activities The Global Health experience consists of a preparatory curriculum, a global health research experience, and presentation of projects and service-learning outcomes upon return from the research/project experience. ♦ Mentored by an interdisciplinary International Committee that has established a large network of global health project and research opportunities around the world, more than 30% of WCMC students participate in these activities. (http://weill.cornell.edu/globalhealth/contact/global_health_education_contact/) WCCC The Weill Cornell Community Clinic was established by the student body in 2008 as a partnership with community social service agencies to provide free health care for adults who are uninsured or underinsured. ♦ Student participation is robust as students undertake the administration and patient care activities under the mentorship of a faculty member and a Steering Committee. (http://weill.cornell.edu/education/admissions/wc_comm_clinic.html) Music and Medicine ♦ A new initiative, with partners including The Julliard School, the 92nd Street Y, and Carnegie Hall, this program seeks to enable medical students to continue a musical life while in medical school, provide access to rehearsal space and musical mentors, and explore the role of music in healing, academic, and research programs.

Pilot of Direct Observation of Clinical Skills (DOCS) in a Medicine Clerkship: Feasibility and Relationship to Clinical Performance Measures

Purpose: To assess the feasibility of Direct Observation of Clinical Skills (DOCS), a program for formative assessment of students’ clinical skills during a medicine clerkship and to determine relationships between DOCS measures and other clinical performance measures.Method: From August, 2004 through June, 2005, Medicine Clerks assigned to the primary on-campus clinical site were asked to participate in the pilot phase of the DOCS program. Students were asked to complete at least one DOCS exercise focused on interviewing, physical examination, or oral case presentation.Results: Of the 79 students who rotated on the Medicine Clerkship during the pilot period, 79% (n=62) participated in DOCS, and 163 forms were submitted for evaluation. Seventy-seven percent (77%) of the clinical observations occurred while on-call or during daily rounds. Seventy-three (73%) of observations were completed in 30 minutes or less. In 89% of encounters students received at least 5 minutes of verbal feedback. Satisfaction ratings from both students and observers were ’’moderately satisfied’’ or better. Global ratings from DOCS physical exam and case presentation sections were strongly correlated with both faculty ratings of clinical performance and final clerkship grade. DOCS measures were not statistically related to clerkship written examination scores.Conclusions: These data support the feasibility of the DOCS session for formative assessment of student interviewing, physical examination, and oral case presentation skills during a medicine clerkship. Observer ratings from DOCS physical examination and case presentation sections were found to be predictors of final clerkship grade.

Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees

Direct observation of medical trainees with actual patients is important for performance-based clinical skills assessment. Multiple tools for direct observation are available, but their characteristics and outcomes have not been compared systematically. To identify observation tools used to assess medical trainees' clinical skills with actual patients and to summarize the evidence of their validity and outcomes. Electronic literature search of PubMed, ERIC, CINAHL, and Web of Science for English-language articles published between 1965 and March 2009 and review of references from article bibliographies. Included studies described a tool designed for direct observation of medical trainees' clinical skills with actual patients by educational supervisors. Tools used only in simulated settings or assessing surgical/procedural skills were excluded. Of 10 672 citations, 199 articles were reviewed and 85 met inclusion criteria. Two authors independently abstracted studies using a modified Best Evidence Medical Education coding form to inform judgment of key psychometric characteristics. Differences were reconciled by consensus. A total of 55 tools were identified. Twenty-one tools were studied with students and 32 with residents or fellows. Two were used across the educational continuum. Most (n = 32) were developed for formative assessment. Rater training was described for 26 tools. Only 11 tools had validity evidence based on internal structure and relationship to other variables. Trainee or observer attitudes about the tool were the most commonly measured outcomes. Self-assessed changes in trainee knowledge, skills, or attitudes (n = 9) or objectively measured change in knowledge or skills (n = 5) were infrequently reported. The strongest validity evidence has been established for the Mini Clinical Evaluation Exercise (Mini-CEX). Although many tools are available for the direct observation of clinical skills, validity evidence and description of educational outcomes are scarce.

Pilot of Direct Observation of Clinical Skills (DOCS) in a Medicine Clerkship:Feasibility and Relationship to Clinical Performance Measures

Purpose:To assess the feasibility of Direct Observation of Clinical Skills (DOCS), a program for formative assessment of students’ clinical skills during a medicine clerkship and to determine relationships between DOCS measures and other clinical performance measures.Method:From August, 2004 through June, 2005, Medicine Clerks assigned to the primary on-campus clinical site were asked to participate in the pilot phase of the DOCS program. Students were asked to complete at least one DOCS exercise focused on interviewing, physical examination, or oral case presentation.Results:Of the 79 students who rotated on the Medicine Clerkship during the pilot period, 79% (n = 62) participated in DOCS, and 163 forms were submitted for evaluation. Seventy-seven percent (77%) of the clinical observations occurred while on-call or during daily rounds. Seventy-three (73%) of observations were completed in 30 minutes or less. In 89% of encounters students received at least 5 minutes of verbal feedback. Satisfaction ratings from both students and observers were “moderately satisfied” or better. Global ratings from DOCS physical exam and case presentation sections were strongly correlated with both faculty ratings of clinical performance and final clerkship grade. DOCS measures were not statistically related to clerkship written examination scores.Conclusions:These data support the feasibility of the DOCS session for formative assessment of student interviewing, physical examination, and oral case presentation skills during a medicine clerkship. Observer ratings from DOCS physical examination and case presentation sections were found to be predictors of final clerkship grade.

Pilot of Direct Observation of Clinical Skills (DOCS) in a Medicine Clerkship: Feasibility and Relationship to Clinical Performance Measures

Pilot of Direct Observation of Clinical Skills (DOCS) in a Medicine Clerkship: Feasibility and Relationship to Clinical Performance Measures

Achieving effective supervision.

Supervision probably does have benefits both for the maintenance and improvement of clinical skills and for job satisfaction, but the data are very thin and almost non-existent in the area of alcohol and other drugs services. Because of the potential complexity of objectives and roles in supervision, a structured agreement appears to be an important part of the effective supervision relationship. Because sessions can degenerate easily into unstructured socialization, agendas and session objectives may also be important. While a working alliance based on mutual respect and trust is an essential base for the supervision relationship, procedures for direct observation of clinical skills, demonstration of new procedures and skills practice with detailed feedback appear critical to supervision's impact on practice. To ensure effective supervision, there needs not only to be a minimum of personnel and resources, but also a compatibility with the values and procedures of management and staff, access to supervision training and consultation and sufficient incentives to ensure it continues.